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WHAT IS CLAIMED IS: 

xxx 1 . A method of processing input data defining (i) the positions of features in a 
sequence of images of at least one object which represent features on the object, 
(ii) an estimate of a respective camera projection for each image defining the 
projection of points on the object into the image, and (iii) 3D feature points 
comprising estimates of the positions in three-dimensions of features on the object 
represented by the features in the images, to generate output data defining further 
estimates of the camera projections, the method comprising processing respective 
subsets of the images in the sequence by: 

calculating an error for the camera projections of the images in the subset 
by projecting selected 3D feature points into the images in the subset using the 
camera projections of the images in the subset and determining the distance 
between the positions of the projected points and the positions of the corresponding 
features in the images; and 

calculating changed positions for at least some of the selected 3D feature 
points and changed camera projections for the images in the subset by using the 
positions in the images in the subset of the features which correspond to the 
selected 3D feature points to determine changed positions for at least some of the 
selected 3D feature points and changed camera projections for the images in the 
subset which reduce the calculated error, 

wherein the respective subsets of images are selected sequentially from the 
sequence so that each image in the sequence is processed in a subset at least once. 

2. A method according to claim 1, wherein the changed positions for the 3D 
feature points and the changed camera positions are calculated by reducing the 
calculated error by performing processing in accordance with a minimisation 
procedure to change iteratively the positions of at least some of the selected 3D 
feature points and the camera projections of the images in the subset and to 
calculate an updated error by projecting the 3D feature points from their changed 
positions into the images in the subset using the changed camera projections of the 
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images in the subset, and determining the distance between the positions of the 
projected points and the positions of the corresponding features in the images. 

3. A method according to claim 2, wherein the calculated error is reduced by 
performing processing in accordance with a non-linear least squares minimisation 
procedure. 

4. A method according to claim 3, wherein the calculated error is reduced by 
performing processing in accordance with a Levenberg-Marquardt minimisation 
procedure. 

5. A method according to claim 1, wherein the changed positions for the 3D 
feature points and the changed camera projections are calculated by decomposing a 
matrix representing the positions in the images in the subset of the features which 
correspond to the 3D feature points to be changed into the product of a first matrix 
representing the changed camera projections of the images in the subset and a 
second matrix representing the changed positions of the 3D feature points. 

6. A method according to claim 1, wherein the selected 3D feature points used 
to calculate an. error comprise every 3D feature point which corresponds to a 
feature having a measured position in at least one of the images in the subset being 
processed. 

7. A method according to claim 1, wherein the 3D feature points for which 
changed positions are calculated comprise every 3D feature point which has a 
measured position in at least one of the images in the subset being processed but no 
measured position in any of the other images in the sequence. 

Claim 8. A method according to claim 1, wherein each respective subset 

contains the same number of images. 
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9. A method according to claim 1 , wherein the number of images in a subset 
is set in dependence upon the number of features in the images having a position 
defined in the data to be processed. 

10. A method according to claim 1, wherein the respective subsets of images 
are selected sequentially from the sequence with an overlap. 

11. A method according to claim 1 , further comprising the step of generating a 
signal conveying the further estimates of the camera projections. 

12. A method according to claim 1 1, further comprising the step of recording 
the signal either directly or indirectly. 

13. A method according to claim 1, further comprising the step of processing 
image data defining the images in the sequence to generate the input data. 

14. A method according to claim 1 , further comprising the step of using the 
further estimates of the camera projections to generate data defining a 3D computer 
model of the scene in the images. 

15. A method according to claim 14, further comprising the step of generating a 
signal conveying the 3D computer model. 

16. A method according to claim 1 5, further comprising the step of recording 
the signal either directly or indirectly. 

1 7. Apparatus for processing input data defining (i) the positions of features in 
a sequence of images of at least one object which represent features on the object, 
(ii) an estimate of a respective camera projection for each image defining the 
projection of points on the object into the image, and (iii) 3D feature points 
comprising estimates of the positions in three-dimensions of features on the object 
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represented by the features in the images, to generate output data defining further 
estimates of the camera projections, comprising a processor for processing 
respective subsets of the images in the sequence by: 

calculating an error for the camera projections of the images in the subset 
by projecting selected 3D feature points into the images in the subset using the 
camera projections of the images in the subset and determining the distance 
between the positions of the projected points and the positions of the corresponding 
features in the images; and 

calculating changed positions for at least some of the selected 3D feature 
points and changed camera projections for the images in the subset by using the 
positions in the images in the subset of the features which correspond to the 
selected 3D feature points to determine changed positions for at least some of the 
selected 3D feature points and changed camera projections for the images in the 
subset which reduce the calculated error, 

wherein the processor is arranged to process respective subsets of images 
from the sequence sequentially such that each image in the sequence is processed 
in a subset at least once. 

18. Apparatus according to claim 17, wherein the processor is arranged to 
calculate the changed positions for the 3D feature points and the changed camera 
positions by reducing the calculated error by performing processing in accordance 
with a minimisation procedure to change iteratively the positions of at least some 
of the selected 3D feature points and the camera projections of the images in the 
subset and to calculate an updated error by projecting the 3D feature points from 
their changed positions into the images in the subset using the changed camera 
projections of the images in the subset, and determining the distance between the 
positions of the projected points and the positions of the corresponding features in 
the images. 
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19. Apparatus according to claim 18, wherein the processor is arranged to 
reduce the calculated error by performing processing in accordance with a non- 
linear least squares minimisation procedure. 

20. Apparatus according to claim 19, wherein the processor is arranged to 
reduce the calculated error by performing processing in accordance with a 
Levenberg-Marquardt minimisation procedure. 

21. Apparatus according to claim 17, wherein the processor is arranged to 
calculate the changed positions for the 3D feature points and the changed camera 
projections by decomposing a matrix representing the positions in the images in the 
subset of the features which correspond to the 3D feature points to be changed into 
the product of a first matrix representing the changed camera projections of the 
images in the subset and a second matrix representing the changed positions of the 
3D feature points. 

22. Apparatus according to claim 17, wherein the processor is arranged to 
perform processing so that the selected 3D feature points used to calculate an error 
comprise every 3D feature point which corresponds to a feature having a measured 
position in at least one of the images in the subset being processed. 

23. Apparatus according to claim 17, wherein the processor is arranged to 
perform processing so that the 3D feature points for which changed positions are 
calculated comprise every 3D feature point which has a measured position in at 
least one of the images in the subset being processed but no measured position in 
any of the other images in the sequence. 

24. Apparatus according to claim 17, wherein the processor is arranged to 
perform processing so that each respective subset contains the same number of 
images. 
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25. Apparatus according to claim 17, wherein the processor is arranged to 
perform processing so that the number of images in a subset is set in dependence 
upon the number of features in the images having a position defined in the data to 
be processed. 

26. Apparatus according to claim 17, wherein the processor is arranged to 
perform processing so that the respective subsets of images are selected 
sequentially from the sequence with an overlap. 

27. Apparatus according to claim 17, further comprising an input data generator 
for processing image data defining the images in the sequence to generate the input 
data. 

28. Apparatus according to claim 17, further comprising a 3D computer model 
data generator for using the further estimates of the camera projections to generate 
data defining a 3D computer model of the scene in the images. 

29. A storage device storing instructions for causing a programmable 
processing apparatus to become operable to perform a method as set out in at least 
one of claims 1 to 16. 

30. A signal conveying instructions for causing a programmable processing 
apparatus to become operable to perform a method as set out in at least one of 
claims 1 to 16. 

31. In an image processing method for processing image data defining a 
sequence of images of at least one object to generate a three-dimensional computer 
model of the object by matching features in the images, using the matching features 
to determine camera projections defining the projection of the object surface into 
the images, and using the matching features and the calculated camera projections 
to generate data defining the three-dimensional computer model of the object, an 
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improvement comprising performing the step of using the matching features to 
determine the camera projections by processing data defining (i) the positions of 
features in a sequence of images of the object which represent features on the 
object, (ii) an estimate of a respective camera projection for each image defining 
the projection of the object into the image, and (iii) 3D feature points comprising 
estimates of the positions in three-dimensions of the features on the object 
represented by the features in the images, to generate data defining further 
estimates of the camera projections, the processing comprising processing 
respective subsets of the images in the sequence such that: 

an error is calculated for the camera projections of the images in the subset 
by projecting selected 3D feature points into the images in the subset using the 
camera projections of the images in the subset to give projected points and 
determining the distance between the positions of the projected points and the 
positions of the corresponding features in the images; and 

changed positions for at least some of the selected 3D feature points and 
changed camera projections for the images in the subset are calculated by using the 
. positions in the images in the subset of the features which correspond to the 
selected 3D feature points to determine changed positions for at least some of the 
selected 3D feature points and changed camera projections for the images in the 
subset which reduce the calculated error, 

wherein the respective subsets of images are selected sequentially from the 
sequence so that each image in the sequence is processed in a subset at least once. 

32. In an image processing apparatus for processing image data defining a 
sequence of images of at least one object to generate a three-dimensional computer 
model of the object by matching features in the images, using the matching features 
to determine camera projections defining the projection of the object surface into 
the images, and using the matching features and the calculated camera projections 
to generate data defining the three-dimensional computer model of the object, a 
method of performing the processing to determine the camera projections by 
processing data defining (i) the positions of features in a sequence of images of at 
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least the object which represent features on the object, (ii) an estimate of a 
respective camera projection for each image defining the projection of the object 
surface into the image, and (iii) 3D feature points comprising estimates of the 
positions in three-dimensipns of the features on the object represented by the 
features in the images, to generate data defining improved estimates of the camera 
projections, the processing comprising processing respective subsets of the images 
in the sequence by: 

calculating an error for the camera projections of the images in the subset 
by projecting selected 3D feature points into the images in the subset using the 
camera projections of the images in the subset to give projected points and 
determining the distance between the positions of the projected points and the 
positions of the corresponding features in the images; and 

calculating changed positions for at least some of the selected 3D feature 
points and changed camera projections for the images in the subset by using the 
positions in the images in the subset of the features which correspond to the 
selected 3D feature points to determine changed positions for at least some of the 
selected 3D feature points and changed camera projections for the images in the 
subset which reduce the calculated error, 

wherein the respective subsets of images are selected sequentially from the 
sequence so that each image in the sequence is processed in a subset at least once. 

33. In an image processing apparatus having a processor for processing image 
data defining a sequence of images of at least one object to generate a three- 
dimensional computer model of the object by matching features in the images, 
using the matching features to determine camera projections defining the 
projection of the object surface into the images, and using the matching features 
and the calculated camera projections to generate data defining the three- 
dimensional computer model of the object, an improvement wherein the processor 
is arranged to use the matching features to determine the camera projections by 
processing data defining (i) the positions of features in a sequence of images of the 
object which represent features on the object, (ii) an estimate of a respective 
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camera projection for each image defining the projection of the object into the 
image, and (iii) 3D feature points comprising estimates of the positions in three- 
dimensions of the features on the object represented by the features in the images, 
to generate data defining further estimates of the camera projections, the processing 
performed by the processor comprising processing respective subsets of the images 
in the sequence such that: 

an error is calculated for the camera projections of the images in the subset 
by projecting selected 3D feature points into the images in the subset using the 
camera projections of the images in the subset to give projected points and 
determining the distance between the positions of the projected points and the 
positions of the corresponding features in the images; and 

changed positions for at least some of the selected 3D feature points and 
changed camera projections for the images in the subset are calculated by using the 
positions in the images in the subset of the features which correspond to the 
selected 3D feature points to determine changed positions for at least some of the 
selected 3D feature points and changed camera projections for the images in the 
subset which reduce the calculated error, 

wherein the respective subsets of images are selected sequentially from the 
sequence so that each image in the sequence is processed in a subset at least once. 

34. Aapparatus for processing input data defining (i) the positions of features in 
a sequence of images of at least one object which represent features on the object, 
(ii) an estimate of a respective camera projection for each image defining the 
projection of points on the object into the image, and (iii) 3D feature points 
comprising estimates of the positions in three-dimensions of features on the object 
represented by the features in the images, to generate output data defining further 
estimates of the camera projections, comprising processing means for processing 
respective subsets of the images in the sequence by: 

calculating an error for the camera projections of the images in the subset 
by projecting selected 3D feature points into the images in the subset using the 
camera projections of the images in the subset and determining the distance 
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between the positions of the projected points and the positions of the corresponding 
features in the images; and 

calculating changed positions for at least some of the selected 3D feature 
points and changed camera projections for the images in the subset by using the 
positions in the images in the subset of the features which correspond to the 
selected 3D feature points to determine changed positions for at least some of the 
selected 3D feature points and changed camera projections for the images in the 
subset which reduce the calculated error, 

wherein the processing means is arranged to process respective subsets of 
images from the sequence sequentially such that each image in the sequence is 
processed in a subset at least once. 

35. A method of processing data defining matching features in images within a 
sequence of images of at least one object to determine a respective camera 
projection for each of at least some of the images defining the projection of points 
on the object surface into the image, the method comprising: 

a selecting step comprising using the matching features to select a first 
image in the sequence and a second image in the sequence for each of a plurality of 
sets of images to give initial end images in the sequence for each set; 

a set forming step comprising, for the selected pair of end images for each 
set, using the matching features to calculate a camera projection for one of the 
images in the pair relative to the other image in the pair, to form a* set thereform; 

a set expanding step comprising increasing the number of images in at least 
some of the sets by using the matching features to calculate a respective camera 
projection for each of at least some of the images lying in the sequence between 
end ynages of a set and each of at least some of the images lying in the sequence 
between consecutive sets, each camera projection being calculated relative to the 
images in a set; and 

a set merging step comprising using the matching features to determine the 
relationship between the camera projections calculated for the different sets, 
thereby to merge the sets. 
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36. A method according to claim 35, wherein, in the selecting step, an image to 
form an end image is selected in dependence upon the number of matching features 
in the image which are also present in the preceding end image in the sequence. 

37. A method according to claim 35, wherein, in the set expanding step, the 
camera projections for images lying between the end images for a given set are 
calculated before the camera projections for images lying between the given set 
and adjacent sets in the sequence are calculated. 

38. A method according to claim 35, wherein, in the step of calculating camera 
projections for images lying between consecutive sets: 

(i) the selected end images are prioritised in dependence upon an 
accuracy of the calculated camera projections of the end images; 

(ii) the end image with the highest priority is selected; 

(iii) the camera projection for the image in the sequence adjacent the 
selected end image is calculated relative to the selected end image to form a new 
end image; 

(iv) the priority of the end images is updated in dependence upon the 
accuracy of the camera projection calculated in step (iii); and 

(v) steps (ii) to (iv) are repeated. 

39. A method according to claim 35, wherein pairs of end images selected in 
the selecting step are considered in turn and, for a given pair of end images, said 
set forming step to form a set therefrom and the set expanding step to calculate a 
respective camera projection for at least one image lying in the sequence between 
the pair of end images and at least one image lying in the sequence between an end 
image of the pair and an end image for a consecutive set, are performed before 
performing the set forming step and the set expanding step for the next pair of end 
images to be considered. 
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40. A method according to claim 35, further comprising the step of generating 
signal conveying the calculated camera projections. 

41. A method according to claim 40, further comprising the step of recording 
the signal either directly or indirectly. 

42. A method according to claim 35, wherein the sequence of images is a 
temporal sequence in which the images are arranged in time-of-recording order. 

43. A method according to claim 35, wherein the images comprise video 
images. 

44. A method according to claim 35, further comprising the step of processing 
image data defining the images in the sequence to generate the data defining the 
matching features. 

45. A method according to claim 35, further comprising the step of using the 
matching features and the calculated camera projections to generate data defining ; 
3D computer model of the object in the images. 

46. A method according to claim 45, further comprising the step of generating 
signal conveying the 3D computer model. 

47. A method according to claim 46, further comprising the step of recording 
the signal either directly or indirectly. 

48. A method of processing data defining matching features in images within £ 
sequence of images of at least one object to determine a respective camera 
projection for each of at least some of the images defining a projection of the 
object into the image, the method comprising: 
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selecting a plurality of groups of the images, each group comprising at least 
two images and at least one pair of consecutive groups in the sequence having 
images in the sequence therebetween which are not part of a group; 

forming a set of images from each selected group by calculating a 
respective camera projection for each image in the group relative to a reference 
image in the group; 

calculating a respective camera projection for each of at least some of the 
images in the sequence not selected to form a group relative to a reference image in 
a group; and 

calculating the relationship between the camera projections calculated for 
each group. 

49. Apparatus for processing data defining matching features in images within 
a sequence of images of at least one object to determine a respective camera 
projection for each of at least some of the images defining the projection of points 
on the object surface into the image, comprising: 

an image selector for using the matching features to select a first image in 
the sequence and a second image in the sequence for each of a plurality of sets of 
images to give initial end images in the sequence for each set; 

a set former for using the matching features to calculate, for the selected 
pair of end images for each set, a camera projection for one of the images in the 
pair relative to the other image in the pair, to form a set therefrom; 

a set expander for increasing the number of images in at least some of the 
sets by using the matching features to calculate a respective camera projection for 
each of at least some of the images lying in the sequence between end images of a 
set and each of at least some of the images lying in the sequence between 
consecutive sets, each camera projection being calculated relative to the images in 
a set; and 

a set merger for using the matching features to determine the relationship 
between the camera projections calculated for the different sets, thereby to merge 
the sets. 
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50. Apparatus according to claim 49, wherein the image selector is arranged to 
select an image to form an end image in dependence upon the number of matching 
features in the image which are also present in the preceding end image in the 
sequence. 

51. Apparatus according to claim 49, wherein the set expander is arranged to 
calculate the camera projections for images lying between the end images for a 
given set before calculating the camera projections for images lying between the 
given set and adjacent sets in the sequence. 

52. Apparatus according to claim 49, wherein the set expander is arranged to 
calculate camera projections for images lying between consecutive sets by: 

(i) prioritising the selected end images in dependence upon an 
accuracy of the calculated camera projections of the end images; 

(ii) selecting the end image with the highest priority; 

(iii) calculating the camera projection for the image in the sequence 
adjacent the selected end image relative to the selected end image to form a new 
end image; 

(iv) updating the priority of the end images in dependence upon the 
accuracy of the camera projection calculated in step (iii); and 

(v) repeating steps (ii) to (iv). 

53. Apparatus according to claims 49, wherein the apparatus is arranged to 
operate so that pairs of end images selected by the image selector are considered in 
turn and, for a given pair of end images, the set former is arranged to form a set 
therefrom and the set expander is arranged to calculate a respective camera 
projection for at least one image lying in the sequence between the pair of end 
images and at least one image lying in the sequence between an end image of the 
pair and an end image for a consecutive set, before forming a set and expanding the 
set for the next pair of end images to be considered. 
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54. Apparatus according to claim 49, wherein the sequence of images is a 
temporal sequence in which the images are arranged in time-of-recording order. 

55. Apparatus according to claim 49, wherein the images comprise video 
images. 

56. Apparatus according to claim 49, further comprising a feature matcher for 
processing image data defining the images in the sequence to generate the data 
defining the matching features. 

57. Apparatus according to claim 49, further comprising a 3D computer model 
data generator for using the matching features and the calculated camera 
projections to generate data defining a 3D computer model of the object in the 
images. 

58. Apparatus for processing data defining matching features in images within 
a sequence of images of at least one object to determine a respective camera 
projection for each of at least some of the images defining a projection of the 
object into the image, comprising: 

an image group calculator for selecting a plurality of groups of the images, 
each group comprising at least two images and at least one pair of consecutive 
groups in the sequence having images in the sequence therebetween which are not 
part of a group; 

an image set calculator for forming a set of images from each selected 
group by calculating a respective camera projection for each image in the group 
relative to a reference image in the group; 

a camera projection calculator for calculating a respective camera 
projection for each of at least some of the images in the sequence not selected to 
form a group relative to a reference image in a group; and 

a camera projection relationship calculator for calculating the relationship 
between the camera projections calculated for each group. 
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59. A storage device storing instructions for causing a programmable 
processing apparatus to become operable to perform a method as set out in at least 
one of claims 35 to 48. 

60. A signal conveying instructions for causing a programmable processing 
apparatus to become operable to perform a method as set out in at least one of 
claims 35 to 48. 

61 . In an image processing method for processing image data defining a 
sequence of images of at least one object to generate a three-dimensional computer 
model of the object by matching features in the images, using the matching features 
to determine camera projections defining the projection of the object surface into 
the images, and using the matching features and the calculated camera projections 
to generate data defining the three-dimensional computer model of the object, an 
improvement comprising performing the step of using the matching features to 
determine the camera projections by: 

using the matching features to select a first image in the sequence and a 
second image in the sequence for each of a plurality of sets of images to give initial 
end images in the sequence for each set; 

for the selected pair of end images for each set, using the matching features 
to calculate a camera projection for one of the images in the pair relative to the 
other image in the pair, to form a set therefrom; 

increasing the number of images in at least some of the sets by using the 
matching features to calculate a respective camera projection for each of at least 
some of the images lying between pairs of end images in the sequence and each of 
at least some of the images lying between consecutive sets in the sequence, each 
camera projection being calculated relative to the images in a set; and 

using the matching features to determine the relationship between the 
camera projections calculated for the different sets, thereby to merge the sets. 
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62. In an image processing apparatus for processing image data defining a 
sequence of images of at least one object to generate a three-dimensional computer 
model of the object by matching features in the images, using the matching features 
to determine camera projections defining the projection of the object into the 
images, and using the matching features and the calculated camera projections to 
generate data defining the three-dimensional computer model of the object, a 
method of performing the processing to determine the camera projections by: 

selecting a plurality of groups of the images, each group comprising at least 
two images and at least one pair of consecutive groups in the sequence having 
images in the sequence therebetween which are not part of a group; 

forming a set of images from each selected group by calculating a 
respective camera projection for each image in the group relative to a reference 
image in the group; 

calculating a respective camera projection for each of at least some of the 
images in the sequence not selected to form a group relative to a reference image in 
a group; and 

calculating the relationship between the camera projections calculated for 
each group. 

63. In an image processing apparatus having a processor for processing image 
data defining a sequence of images of at least one object to generate a three- 
dimensional computer model of the object by matching features in the images, 
using the matching features to determine camera projections defining the 
projection of the object surface into the images, and using the matching features 
and the calculated camera projections to generate data defining the three- 
dimensional computer model of the object, wherein the processor is arranged to use 
the matching features to determine the camera projections by: 

using the matching features to select a first image in the sequence and a 
second image in the sequence for each of a plurality of sets of images to give initial 
end images in the sequence for each set; 
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for the selected pair of end images for each set, using the matching features 
to calculate a camera projection for one of the images in the pair relative to the 
other image in the pair, to form a set therefrom; 

increasing the number of images in at least some of the sets by using the 
matching features to calculate a respective camera projection for each of at least 
some of the images lying between pairs of end images in the sequence and each of 
at least some of the images lying between consecutive sets in the sequence, each 
camera projection being calculated relative to the images in a set; and 

using the matching features to determine the relationship between the 
camera projections calculated for the different sets, thereby to merge the sets. 

64. An apparatus for processing data defining matching features in images 
within a sequence of images of at least one object to determine a respective camera 
projection for each of at least some of the images defining the projection of points 
on the object surface into the image, comprising: 

selecting means for using the matching features to select a first image in the 
sequence and a second image in the sequence for each of a plurality of sets of 
images to give initial end images in the sequence for each set; 

set forming means for using the matching features to calculate, for the 
selected pair of end images for each set, a camera projection for one of the images 
in the pair relative to the other image in the pair, to form a set therefrom; 

set expanding means for increasing the number of images in at least some 
of the sets by using the matching features to calculate a respective camera 
projection for each of at least some of the images lying in the sequence between 
end images of a set and each of at least some of the images lying in the sequence 
between consecutive sets, each camera projection being calculated relative to the 
images in a set; and 

set merging means for using the matching features to determine the 
relationship between the camera projections calculated for the different sets, 
thereby to merge the sets. 
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65. An apparatus for processing data defining matching features in images 
within a sequence of images of at least one object to determine a respective camera 
projection for each of at least some of the images defining a projection of the 
object into the image, comprising: 

means for selecting a plurality of groups of the images, each group 
comprising at least two images and at least one pair of consecutive groups in the 
sequence having images in the sequence therebetween which are not part of a 
group; 

means for forming a set of images from each selected group by calculating 
a respective camera projection for each image in the group relative to a reference 
image in the group; 

means for calculating a respective camera projection for each of at least 
some of the images in the sequence not selected to form a group relative to a 
reference image in a group; and 

means for calculating the relationship between the camera projections 
calculated for each group. 



